Repository navigation
Bump data substrate to payload reservation (tx_service#578); add checkpoint-cycle multi-node test - #507
Bump data substrate to payload reservation (tx_service#578); add checkpoint-cycle multi-node test#507liangjchen wants to merge 1 commit into
Conversation
…i-node test Moves the data substrate submodule to eloqdata/tx_service#578: write-lock acquisitions reserve their payload on the destination shard before the lock attempt, and a request parked for shard memory is rejected with OUT_OF_MEMORY after a configured number of concluded checkpoint cycles. EloqDoc needs no source change for it. Adds cmake/tests/multinode/ckpt_cycles.py, which checks the per-node count of concluded checkpoint cycles on a live cluster through the checkpoint_cycles_concluded_total metric: rounds merged into a held flush conclude only when it lands, one each, and a round whose node group changes leader mid-round still concludes (cloud mode). The README lists the new test, and voter_takeover.py, which it had left out. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. WalkthroughAdds a multinode test runner for checkpoint-cycle counts. It covers deduplicated data-sync rounds and cloud-mode voter takeover, documents the scenarios, and updates the data substrate subproject reference. ChangesMultinode checkpoint-cycle tests
Data substrate reference
Estimated code review effort: 3 (Moderate) | ~25 minutes Change: Other Merge Risk:
|
Summary
main(at Change data capture for EloqDoc: log scan, schema-op timestamps, cluster gate, multi-node fixes tx_service#577) to Reserve payload memory before write-lock acquisition; bound memory waits by checkpoint cycles tx_service#578.cmake/tests/multinode/ckpt_cycles.pyfor the checkpoint-cycle count that #578 introduces.cmake/tests/multinode/README.md: lists the new test, andvoter_takeover.py, which it had left out.EloqDoc needs no source change for the bump.
What the bump brings
Details are in eloqdata/tx_service#578. In short:
OUT_OF_MEMORYaftershard_memory_wait_ckpt_cyclesconcluded checkpoint cycles (default 5). Before, most requests waited indefinitely. Log replay and standby forwarding are exempt.checkpoint_cycles_concluded_total.Visible to EloqDoc users: under sustained memory pressure a write or read can now fail with an out-of-memory error where it used to wait.
The new test
ckpt_cycles.pyreads each node'scheckpoint_cycles_concluded_totalon a live cluster and checks two cases that unit tests cannot stage:Like the other scripts in that directory, it is run by hand against a server built with fault points; it is not registered in CI.
Test plan
Run on 2026-11-13 inside the CI image, on EloqDoc
mainplus this PR's two files, with the data substrate at the #578 change (RelWithDebInfo, EloqStore data store, RocksDB log state, fault injection on):cmake --build <dir> --target eloqdoc eloqdoc-tests host_managerctest --test-dir <dir> --label-exclude '^integration$': passedeloqdoc-server-smoke,eloqdoc-ddl-recovery,eloqdoc-cluster-gate,eloqdoc-change-stream-drivers,eloqdoc-cdc-consumer: all passedsmoke,dss_starting,replay_rerequest,ckpt_cycles,cdc_step_down,cluster_gate_multinode,ddl_split_recovery: 7 of 7 passedvoter_takeoverplus the same 7: 7 of 8 passed.ddl_split_recoveryhung in its last scenario and passed on a rerun. The cause is a race that exists onmainand is fixed separately in Fix a recovered range split dropped at replay finish: read leader and candidate terms as one pair tx_service#579; it is not related to this change.Not run: the js suites, and any performance comparison.
The data substrate tree that was tested differs from the pinned commit by one re-indented line.
Before merging
The submodule points at the head of the #578 branch. tx_service merges by squash, so once #578 is merged this pointer must be moved to the resulting commit on tx_service
main. Please do not merge this PR before that.🤖 Generated with Claude Code
Summary by CodeRabbit